ELSEVIER 


Available  online  at  www.sciencedirect.com 

SciVerse  ScienceDirect 


Procedia  Computer  Science  8  (2012)  165  -  170 


Procedia 

Computer  Science 


New  Challenges  in  Systems  Engineering  and  Architecting 
Conference  on  Systems  Engineering  Research  (CSER) 
2012  -  St.  Louis,  MO 
Cihan  H.  Dagli,  Editor  in  Chief 
Organized  by  Missouri  University  of  Science  and  Technology 


Using  System  Architecture  Maturity  Artifacts  to  Improve 
Technology  Maturity  Assessment 

Matin  Sarfaraz3,  Dr.  Brian  J.  Sauser3,  Edward  W.  Bauerb 

a Stevens  Institute  of  Technology,  Castle  Point  on  Hudson,  Hoboken,  NJ  07030  USA 
bUS  Army  RDECOM-ARDEC,  Picatinny  Arsenal,  Picatinny,  NJ  07806  USA 

Abstract 

The  Technology  Readiness  Level  (TRL)  is  a  measurement  used  to  assess  the  maturity  of  a  technology 
prior  to  its  inclusion  in  a  system.  It  is  a  management  tool  utilized  by  program  managers,  project  managers, 
and  others  in  acquisition  management  to  assess  technology  maturity.  The  TRL  of  a  technology  is 
determined  by  the  assessment  of  Subject  Matter  Experts  (SMEs)  who  examine  the  degree  to  which  a 
criteria  is  being  fulfilled.  One  of  the  current  deficiencies  in  using  TRLs  is  the  subjectivity  in  determining 
the  readiness  value.  This  paper  aims  to  reduce  subjectivity  in  TRL  maturity  assessments  by  utilizing  the 
maturity  artifacts  present  in  system  architecture  models.  This  paper  will  propose  a  technique  and  research 
methodology  that  can  support  TRL  in  technology  maturity  assessment  in  the  design  and  development 
phase  of  a  technology  lifecycle. 
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1.  Introduction 

In  the  area  of  technology  management,  program  managers  suffer  from  schedule  slippages, 
cancellations,  and  failure  to  meet  performance  objectives.  The  GAO  claimed  that  maturing  new 
technology  before  it  is  included  in  a  product  is  perhaps  the  most  determinant  factor  in  the  success  of  the 
eventual  product  [1].  To  that  end,  Technology  Readiness  Level  (TRL)  has  proven  to  be  a  beneficial  metric 
in  assessing  the  risk  associated  with  a  developing  or  acquired  technology.  However,  one  of  the  deficiencies 
in  using  the  TRL  metric  is  that  estimates  of  maturity  are  predominantly  formulated  by  SMEs  [2,  3]. 
Although  there  are  guidelines  and  tools  to  support  the  assessment  process  such  as  the  TRL  calculator  or 
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DoD  Deskbook[4,  5],  the  final  estimation  of  maturity  is  left  to  the  evaluator(s),  who  make  (s)  the  decision 
subjective  [6]. 

It  is  the  goal  of  this  paper  to  present  a  more  informed  decision  making  method  that  will  assist 
managers  in  measuring  and  tracking  the  progress  and  risks  involved  in  maturity  assessment.  To 
accomplish  this,  maturity  artifacts,  the  pieces  of  information  needed  by  decision  makers  to  make  informed 
decisions,  are  identified  and  mapped  to  system  architectural  elements.  Given  the  inaccuracy,  imprecision, 
or  lack  of  knowledge  coupled  with  the  complexity  of  engineering  systems,  this  research  attempts  to 
improve  upon  decision  making  in  the  area  of  maturity  assessment  by  providing  more  information  for  better 
grounded  decisions. 

To  achieve  the  goal  stated  above,  this  paper  explores  the  combined  use  of  the  Department  of 
Defence  Architecture  Framework  (DoDAF)  and  TRL  metric  in  technology  maturity  assessment.  System 
architectures  facilitate  decision  making  by  conveying  the  necessary  information  to  the  decision  maker  by 
presenting  architecture  information,  and  the  TRL  provides  a  metric  to  assess  the  maturity  of  a  technology 
at  any  given  time.  Architecture  data  supports  acquisition  program  management  and  systems  development 
by  representing  system  concepts,  design,  and  implementation  as  they  mature  over  time,  which  enable  and 
support  operational  requirements  [7].  In  the  latest  version  of  DoDAF,  Meta  Model  (DM2)  have  been 
introduced  to  define  concepts  and  models  and  to  support  Defence  Acquisition  System  (DAS)  and 
Planning,  Programming,  Budgeting,  and  Execution  (PPBE),  which  are  only  two  of  the  six  core  processes 
within  DoD [8]. 

This  research  correlates  maturity  artifacts,  the  pieces  of  information  needed  by  decision  makers  to 
make  informed  decision,  to  system  architectural  elements  that  are  present  in  DoDAF  2.0  models.  To 
achieve  this,  this  research  first  identifies  a  set  of  DoDAF  2.0  models  that  are  suitable  for  maturity 
assessment.  The  challenge  to  identify  the  set  of  DoDAF  2.0  models  suitable  for  assessment  becomes  easier 
with  the  use  of  the  TRL  calculator  tool.  This  research  pairs  each  TRL  calculator  question  to  the  best  suited 
model(s)  in  DoDAF  2.0.A  survey  study  will  be  used  to  demonstrate  the  effectiveness  of  this  new  approach 
in  comparison  to  the  current  practice  of  using  TRLs. 

2.  Literature  Review 

The  use  of  technology  maturity  metrics  within  aerospace  has  been  around  since  the  introduction 
of  TRL  in  the  1980’s  [9].  Developed  by  United  States  (US)  National  Aeronautic  and  Space  Administration 
(NASA),  TRL  is  a  nine  level  scale  that  describes  the  maturity  of  a  technology  with  respect  to  a  particular 
use  [10].  Following  its  introduction,  US  government  agencies  (e.g.  Department  of  Defense  (DoD),  Army, 
Department  of  Energy)  and  their  contractors  (the  Sandia  National  Laboratory)  adopted  this  scale  [1,  9]. 
Tan  has  pointed  out  to  the  diverse  ways  that  agencies  and  organizations  have  employed  the  TRL  metric  [6]. 
To  support  the  use  of  TRL,  the  DoD  has  published  Technology  Readiness  Assessment  (TRA)  Deskbooks 
[11,  12].  In  2002,  William  Nolte  at  AFRL  developed  and  released  the  first  TRL  Calculator  (it  is  a 
Microsoft  Excel  application)  for  both  hardware  and  software  [4].  The  TRL  Calculator  attempts  to  fill  for 
the  lack  of  guidance  on  how  to  use  TRLs  by  providing  the  program  manager  with  a  tool  that  can  measure 
the  maturity  of  a  given  technology  [4].  The  TRL  Calculator  tool  provides  a  repeatable  set  of  questions  for 
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determining  a  TRL.  The  TRL  calculator  works  by  answering  a  series  of  checklist  questions.  The  answer  to 
each  question  is  assumed  to  be  in  a  documentable  form  (reports,  CONOPS,  analysis,  chart,  etc.),  and 
assumed  to  be  available  and  collected  prior  to  the  beginning  of  assessment.  In  this  research,  we  assume 
that  information  relating  to  maturity  assessment  can  be  found  in  the  system  architecture. 

While  the  TRL  metric  has  been  sufficient  at  evaluating  technology  readiness,  various  authors 
have  pointed  to  TRLs  deficiencies  [13].  Sauser  et.  al.  pointed  that  TRLs  overlook  integration  between  two 
technologies,  which  led  to  in  the  development  of  the  Integration  Readiness  Level  (IRL)  metric  [14].  In 
addition,  the  TRL  metric  is  a  soft  measure.  Silvestro  states,  Soft  measures  are  those  which  are  qualitative, 
judgmental,  subjective,  and  based  on  perceptual  data  [15].  The  lack  of  formal  method  of  implementing 
TRLs  has  also  contributed  to  claims  of  subjective  assessments,  which  can  be  due  to  biased  technology 
developers  and  the  broad  interpretation  for  the  definitions  of  each  TRL  level  [13].  Despite  these 
deficiencies,  the  TRL  is  the  metric  of  choice  for  DoD  to  guide  a  technology  through  the  lifecycle 
development  phases. 

In  the  mid  1990s,  the  DoD  determined  that  a  common  approach  was  needed  for  describing  its 
architectures,  so  that  DoD  systems  could  efficiently  communicate  and  interoperate  during  joint  and 
multinational  operations  [16].  In  2009,  DoDAF  2.0  was  introduced,  taking  the  focus  away  from  models 
and  placing  emphasis  on  data.  Aside  from  DoDAF  2.0’s  new  features  that  can  help  in  acquisition  processes 
and  technology  management,  researchers  have  studied  using  DoDAF  in  Technology  Management.  Dimov 
[17]  presented  a  architecture -oriented  modelling  approach  to  assist  in  acquisition  systems  for  one  of 
Bulgaria’s  force-management’s  subsystems.  Hughes  from  the  Air  Force  Institute  of  Technology  used  a 
concept  maturity  model  to  help  to  uncover  the  unknowns  that  plague  a  system  development  [18].  He 
suggested  using  maturity  elements  to  assess  and  mature  a  concept  at  a  given  decision  point.  Philips 
introduced  Human  Readiness  Level  to  complement  TRL  in  program  risk  management  structures,  and 
synthesized  the  technical  details  of  Human  View  in  relation  to  DoDAF  [19]. 

DoDAF  2.0  introduced  new  views  (i.e.  PV-2,  SvcV-9,  SV-9)  and  architecture  modelling 
primitives  to  support  collection  or  architecture  content  that  can  be  used  for  maturity  assessments.  Another 
feature  introduced  in  DoDAF  2.0  was  Fit-for-Purpose  (FFP)  models.  FFP  models  are  useful  in  decision 
making,  and  enable  the  architect  to  focus  on  collecting  and  creating  views  that  are  necessary  for  the 
decision  maker’s  requirements,  and  focusing  the  architecture  to  align  to  the  decision-maker’s  needs  [20]. 
The  survey  of  literature  body  shows  a  gap  in  the  application  of  DoDAF  models  in  maturity  assessment 
using  the  TRL  metric.  The  identification  of  models  and  maturity  artifacts  is  advantageous  to  the  practice  of 
systems  engineering,  helping  in  utilizing  systems  architectures  to  make  more  informed  decisions. 

3.  Research  Approach 

This  research  will  propose  a  new  method  for  technology  maturity  assessment.  Once  this  method 
is  developed,  a  survey  study  will  be  used  to  collect  data  for  statistical  analysis.  Subject  Matter  Experts  for 
a  particular  program  will  be  asked  to  rate  the  technology  maturity  using  two  different  methods:  the  new 
method  and  current  TRL  procedure.  The  information  gathered  by  the  survey  study  will  be  used  to 
determine  if  there  is  a  statistically  significant  difference  between  using  this  new  technique  and  the  current 
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practice  of  using  TRLs.  For  this,  the  aggregated  SME  inputs  would  be  used  to  determine  mean  and 
variation  for  each  practice  to  test  the  hypothesis.  If  a  statistically  significant  difference  is  observed  from 
the  collected  data,  then  the  hypothesis  is  accepted. 

The  technique  this  research  uses  to  pair  maturity  elements  to  DoDAF  views  and  artifacts  is 
achieved  in  two  steps.  In  the  first  step,  maturity  elements  derived  from  the  TRL  calculator  checklist  were 
grouped  to  three  categories.  This  aids  in  understanding  the  maturity  element,  but  more  importantly,  this 
step  is  a  necessity  for  step  2.  This  paper  utilizes  Conceptual  Data  Model  (CDM),  which  is  one  of  the  new 
three  levels  of  DoDAF  Meta  Models  (DM2)  introduced  in  DoDAF  2.0.  The  CDM  defines  concepts 
involving  high-level  data  constructs  from  which  Architectural  Descriptions  are  created,  enabling 
executives  and  managers  at  all  levels  to  understand  the  data  basis  of  Architectural  Description  [8].  In 
Figure  1,  key  concepts  are  grouped  into  three  categories  (Ways,  Means,  and  Ends)  to  facilitate  the 
collection  and  usage  of  architecture  related  data. 


Description  of  each  key  concept  as  it  relates  to  Pdaturity  Assessment 
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Figure  1:  Most  popular  DM2  Conceptual  Data  Model  concepts  used  for  categorizing  maturity  artifacts 

In  the  second  step,  the  CDM  is  mapped  to  a  DoDAF  model.  Through  the  use  of  CDM,  we  can 
bridge  between  the  checklist  questions  in  TRL  calculator  and  views  that  support  technology  maturity 
assessment  in  DoDAF.  Figure  2  shows  the  mapping  of  a  CDM  to  DoDAF  views.  Since  each  CDM  is  used 
in  more  than  one  DoDAF  related  model,  the  selection  of  model  could  prove  a  challenge.  Researchers 
familiar  with  DoDAF  may  have  easier  time  mapping  this  step,  but  there  are  plenty  of  publications  and 
reports  that  summarize  the  use  for  DoDAF  related  models,  and  can  assist  in  accomplishing  this 
activity [21]. The  identification  of  maturity  artifacts  and  CDM  can  lead  to  the  appropriate  selection  of 
DoDAF  model(s)  best  suited  for  storing  and  managing  maturity  artifacts.  Following  the  process  above, 
Figure  3  shows  a  sample  maturity  to  DoDAF  model  alignment  for  two  levels  of  the  TRL  Calculator 
checklist. 
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Figure  2:  Mapping  CDM  to  DoDAF  Models [21] 


Figure  3:  Mapping  of  TRL  checklist  to  DoDAF-related  Models 


Delphi  method  is  used  to  collect  and  benefit  from  the  knowledge  of  the  expertise  and  practitioners 
in  this  field.  Delphi  method  originated  in  a  series  of  studies  that  the  RAND  Corporation  conducted  in  the 
1950’s.  The  objective  was  to  develop  a  technique  to  obtain  the  most  reliable  consensus  of  a  group  of 
experts  [22,  23].  Delphi  may  be  characterized  as  a  method  for  structuring  a  group  communication  process 
so  that  the  process  is  effective  in  allowing  a  group  of  individuals,  as  a  whole,  to  deal  with  a  complex 
problem  [22].  A  Delphi  Survey  has  three  main  tasks.  First,  defining  and  describing  the  topic  and  preparing 
questions  to  send  experts;  second,  selection  of  a  panel  of  participating  experts;  and  third,  organizing  and 
running  the  survey[24].  The  input  received  from  the  SME  is  going  to  be  vital  in  a  better  identification  of 
DoDAF  model  suitable  for  assessment,  but  the  development  of  FFP  models  that  can  allow  managers  in 
using  DoDAF  in  program  development. 

4.  Conclusion  and  Future  Works: 

Far  too  often  have  decision  makers  have  been  forced  to  make  decisions  based  on  insufficient  data, 
resulting  in  projects  that  are  over-budget  and  over- schedule.  This  paper  introduces  a  new  technique  that 
incorporates  the  use  of  DM2  CMEs  to  assist  in  the  characterizations  of  maturity  elements.  Given  the 
increase  application  of  system  architectures  in  organizations  and  government  agencies,  this  paper  aims  to 
examine  the  application  of  DoDAF  2.0  in  technology  management.  This  is  possible  by  capturing 
information  through  a  structured  documentation  process.  The  increased  availability  of  data  and  increased 
transparency  among  reviewers  decreases  the  risk  of  uninformed  decision  making  when  available 
information  is  determined  to  be  insufficient  or  missing,  leaving  less  room  for  uninformed  decision  making. 

The  future  works  for  this  research  includes  survey  study,  and  execution  of  Delphi  method  to  collect 
the  knowledge  of  SME  in  better  identifying  and  selecting  DoDAF  models.  In  addition,  Delphi  method  will 
allow  on  the  development  of  FFP  models,  which  is  strongly  encouraged  for  DoDAF  2.0[8].  The  authors  of 
this  research  strongly  recommend  mapping  the  integration  readiness  levels  to  DoDAF  models  as  well, 
which  shall  be  vital  for  more  informed  decision  making  in  system  maturity  assessment. 
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